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study: (1) Why isn't the evaluation process more useful to 
decision-makers and practitioners? and (2) If it is not useful, is it 
worth the effort? In the absence of pertinent literature, research 
focused on people on the "front lines" — the teachers, principals, and 
project implement ers. Central office managers, and evaluators were 
consulted to determine possible communication gaps. Data were 
collected over a 6~year period through formal and informal 
observation of activities and interviews with key actors responsible 
for implementing evaluation unit functions. Results showed that 
program evaluation in schools involves implementing a complex, 
innovative concept that for most school staff (including managers) is 
unfamiiliar, confusing, and even intimidating. This means evaluation 
use is not a simple matter of reviewing recommendations and following 
through ^n suggested changes. In reality, the evaluation use process 
challenges people to modify their routines and design new ways of 
accommodating to changing job demands. After discussing evaluation 
and innovation, the structure of school evaluation use is explained 
and applications are outlined. As with other innovations, integrating 
evaluation into complex school organizations is evolutionary and 
likely to cause disruption. Even so, evaluation is well worth the 
effort. Included are two appendices and a bibliography of 43 
references. (MLH) 
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This paper describes a structure for assessing the school evaluation use 
process that was developed out of a longitudinal case study of 
districtwide and school level evaluation procedures in a large urban 
school district. Two fundamental questions guided our study: 1) If we 
know so much about evaluation procedures, why isn't the evaluation 
process more useful to district level and school site decision makers 
and practitioners? and 2) If it is not useful, is it worth the effort? 

Our intuitive faith in educators and schools, combined with our 
persistent belief in the rational process, drove us to look for the 
logical structure of the process of evaluation use that has apparently 
been missed by evaluation researchers. Our search returned us again and 
again to school people on the "front lines", the teachers, principals 
and project implementers. In addition, we listened closely to central 
office managers to determine where there were communication gaps and, if 
possible, to learn their sources. Data were collected in three separate 
years over a six year period through formal and informal observations of 
activities and interviews with key actors responsible for implementing 
the functions of the districtwide evaluation units during this period. 
The investigation centered on the routine evaluation processes typically 
used in the school system. 

A consistent theme that emerged from our interviews was that program 
evaluation in schools involves implementing a complex, innovative 
concept" that for most school district personnel — even central office 
managers and evaluators Is unfamiliar, confusing, and of ten 
intimidating. This contrasts with our expectation that evaluation use 
is a simple matter of reviewing recommendations and following through on 
suggested changes. In reality, the use of the evaluation process in 
school settings is a challenge requiring people to modify their routines 
and to design new ways of accommodating to job demands that may change 
as a result of the evaluatiLn process. This involves making behavioral 
change at a fundamental level, a level that for most of us is difficult 
to attain. 

Our grounded data pointed to two underlying phenomena. First, we 
observed, as have Lindblom and Cohen (1982), that analytical processes 
that are second nature to evaluators and researchers are quite foreign 
and mysterious to child- and school -ori ented practitioners. A close 
observer of the school district shared an alternative rationality for 
using evaluations that was typical in the system we observed: "The 
implicit purpose of evaluation and testing in this system is to justify 
what you want to do and to eliminate what you don't want to see. It's 
not to solve a problem. It's to meet an agenda. One's frame of 
reference pe'-mits any set of data to be used to make a cogent argument. 
As a result, you can support your own point of view with almost any 
information" (Field Notes, December 6, 1985). 

Second, organizational learning, like human learning, occurs 
developmental ly, requiring adjustment of the operational processes 
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within the organization and, at the same time, basic skills acquisition 

among individual workers (Comfort, 1982), As such, the learning 

process for individuals and organizations is gradual and needs to be 

nurtured, as does any meaningful learning that leads to substantial 
growth and change. 

Evaluation as Innovation 

In examining users' and evaluators' descriptions of their involvement 
with the evaluation process, we began to recognize that we had been 
misinterpreting what appeared be a systematic abuse of the ideal 
evaluation process envisioned by the rational mandates and established 
research traditions (see also Hall, 1982). This year's return to the 
school district, along with our earlier studies (King, Thompson ?jt 
Pechman, 1981; King Sc Pechman, 1982), showed that the process of 
conducting local school evaluations is more baffling to practitioners in 
school districts than we had realized (see also Kennedy, Apling, h 
Newman, 1980). Lack of understanding partly explains the resistance to 
using the available evaluation models that emerged in the heyday of the 
1960s and 1970s (Comfort, 1982; Eash, 1985a; Shapiro, 1985; Stake, 
1985). 

As we looked more closely at our data and listened to the people in 
schools, we began to recognize that conducting and using evaluations in 
school settings constitute a major innovative effort and should be 
analyzed and understood in that light. The hesitant, even obstructive, 
responses of educational systems to innovation are well documented in 
the literature (Comfort, 1982; Havelock, 1971; Popkewitz, Tabachnick, ?x 
Wehlage, 198?^ Wolcott, 1977). A longer look at the use of evaluation 
in the school system revealed that the impact of evaluation mandates 
was, like other major innovations in recent years, very slowly taking 
root. As with the earlier innovations, the evaluation "idea champions" 

in this case the professional evaluation specialists and the 
district's policy makers, top managers and federal program monitors — 
had greater expectations for the evaluation process than did either the 
evaluators or users who had to resolve the practitical day-to-day 
implementation problems, and, not surprisingly, the practitioners' more 
gradual approach controlled what we actually observed in schools. 

The developmental evolution of school innovation adoption has been 
described by Hall and his colleagues at the University of Texas R&D 
Center (Hall, Wallace, & Dossett, 1973; Hall et al., 1975). It begins 
with users' concerns about a particular innovation and involves 
adjusting their feelings, attitudes and worries as a first step in the 
adoption process. As the personal concerns of individual practitioners 
are addressed, they evolve through several stages, from a lack of 
knowledge about the innovation and how to use it to the development of 
skills necessary for the routine use of the innovation. Ultimately, 
the use of the innovation is refined and incorporated into the ongoing 
structure of the system. 



Making Evaluation Worth the Etfort Page 



Implementing evaluations that use rational analytical processes 
requires a good deal ot on-site program staH. For many, it means new 
skills need to be learned and practiced, and there are inevitable 
mistakes to be made. This process is painful, particularly because 
there is not much tolerance for learning by mistake in bureaucratic 
systems. Thus, when the evaluation process is incorporated into a new 
program, its successful use depends on the staff's openness to working 
closely with the process. The ideal of evaluation use envisions school 
site personnel coordinating closely with evaluation teams, applying 
rational and analytical procedures at all levels of instructional 
decision making to support continual school and program improvement, but 
the practice falls far short of the ideal. 

How do we move people and systems beyond their current levels of 
understanding of the evaluation use process? Where do we begin? 

The Structure of School Evaluation Use 

This return after three years to look at the evaluation use process in a 
large school district revealed a more optimistic picture of evaluation 
use in school districts if we expect a developing process, rather than a 
static ideal. Ue can better comprehend apparent failures to use 
evaluation information in predictable ways that adhere to a scientific 
rationality when we see the process as continually evolving, at 
different paces in each program, and m unique patterns depending on the 
individuals responsible for iffipleniention. 

The original naturalistic data from our sample district, gathered 
between 1980 and 1982 (King S( Pechman, 1982; 1984), and this year's 
return to the study site defined the evaluation use problem and allowed 
us to see the underlying rational structure. The evolutionary process 
we observed is described in The Structure of School Evaluation Use 
--SSEU (Appendix A), the framework we developed from our grounded data . 
The SSEU is an instrument that considers school evaluation use as an 
innovation. We combined our findings about the process of evaluation 
use in schools with the way innovations are received as described in 
Hall and Loucks' Levels of Use of Innovations — LoU (see Appendix B; 
Hall, 1982; Hall and Loucks, 1975). Our purpose is to identify 
accurately the evolutionary stages in which the evaluation process is 
implemented so, ultimately, we can help schools use the evaluation 
process more rationally. 

This framework describes the beh. vior of individual users of the 
evaluation process and of evaluation information in schools and school 
systems. It shows the evolutionary structure that users follow in 
developing sophistication in their use of evaluations. Our longitudinal 
data, combined with the observations reported in numt:rous studies of 
school evaluation use, indicated a whole range of responses to the 
process of implementing and using school evaluations (Alkin, Daillak, ?( 
White, 1979; Comfort, 1982; Cooley & Bickel, 1986; Cousins 3( Leithwood, 
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1985; f'lng ?. Pechfnan, 1982). By seeking and defining consistencies in 
the ways practitioners use the evaluation process, we will have a better 
understanding of how to conduct evaluations to assure they respond to 
each user ' s needs. 

The Structure of School Evaluation Use consists of six major levels of 
evaluation use that go from "nonuse" of evaluation to "renewal," We 
examined our three years of field notes and set them against the Loll to 
arrive at the specifications for stages of evaluation use, the "decision 
points" that mark a user's uovement between stages, and the behavioral 
indicators describing the key functions characteristic of users at the 
highest levels of use. Our focus was on the evaluation activity of 
program coordinators and directors, and, only to a limited degree, on 
teachers. 

Nonuse is defined as a preliminary stage in which the user is unaware of 
the potent ial use-fulness of evaluation and the evaluation process and 
does nut look for information to support program implementati on . This 
level of use is not particularly common in schools, since school people 
have become increasingly sensitive to the expectation that schooling 
should be involved in a continuing evaluation process. 

Once potential users take action to learn more about the requirements or 
procedures for conducting an evaluation, they have moved to Stage I, 
Orientation . At this stage, users are exploring the ideas about 
evaluation and are considering how evaluation might fit the needs of 
their own programs. When they i.iitiate the evaluation process by a 
specific action, such as calling for a design or the consultation of an 
evaluator, they have moved to Stage II, Preparation . 

Pr eparati on continues the pr eeval uat i on process. The users may 
recognize the need to collect data for an evaluation but, not knowing 
quite how to do it, they assign the task to a staff person, vaguely 
expecting to receive feedback, but not quite certain what to expect. 
When users take their first steps toward reviewing the data and combine 
that review with a serious effort to use the information, even 
informally and simply, they have moved toward Stages IIIA and Illb, 
Mechani cal Use and Routine Use . 

Mechanical Use and Routine Use are two substages within this phase. 
They are similar, but represent increasingly sophisticated uses of the 
evaluation process. These two levels are the levels of evaluation use 
that we most frequently observed in our study among experienced project 
directors. The latter substage. Routine Use, consists of a more complex 
application of the basic data that is collected in the Mechanical Use 
phase. As in the LoU framework. Routine Use is the minimal desired 
level of achievement. But Routine Use of the educational evaluation 
process is short of the most ideal form of use because it does not reach 
into the daily teaching or managing structure of program implementation. 
Routine Use employs evaluation in a unidimensi onal manner, as an 
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indicator or signal to external sponsors that programs are proceeding 
according to timelines. 

At thB highest levels oi use, there is a continuing evaluation process 
in piacs that is consistent with more rational models oi 
evalt ation-based decision making. Stage IV, Refinement, begins to 
incorporate the use of evaluation information to make substantive 
program changes, but, at this stage, the use of information is 
restricted. This type of evaluation use has been called "charged use" 
like ions in a chemical equation, evaluation information takes on a 
charge — and has the potential to cause a reaction in the system or 
prograo being evaluated (King ?x Pechman, 1982; 1984)- Evaiuators and 
users work collaboratively at this stage and at other higher stages, 
designing evaluations to fit program needs more closely. 

Stage V, Integration , occurs when a user tries to make more specific 
program changes in direct response to the evaiuators' feedback- The 
entire process of evaluation, frotn its inception through the 
systematically planned use of evaluation, is better coordinated than in 
earlier stages. The program coordinator designs the prog»^am to 
incorporate evaluation throughout, but has not yet succeeded in fully 
institutionalizing it into the program and still personally guides most 
of the staff's practical applications of the evaluation process- 
Finally, Stage VI, Renewal , describes the user who continually evaluates 
the evaluation process itself to assure it is adequately serving 
programmatic needs and adjusts and modifies the evaluation when 
necessary. The evaluation process is continuous and well integrated 
into the program's structure. It is comprehensive and reaches all 
levels of management, staff and clients. Political realities and 
limitations notwithstanding. Renewal decision makers who can include 
any leader, teachers, project coordinators, principals, district level 
managers, superintendents, and school board members expect evaluation 
informaticn, routinely take action to obtain it, analyze its application 
and practicality, and, in some rational way, use it- 
Data from our study of this sample district indicated an interesting 
pattern of evaluation use. There were many users, especially new 
principals and teachers, at the Orientation and Preparation levels and 
only a few users were at the higher levels. The actions of principals 
and directors or coordinators of special programs programs typically ^it 
the Preparation sLage and did not reach Mechanical Use except in cases 
of project directors who had lengthy experience in federal programs. 
Nevertheless, arrival at the stage of Preparation shows that there has 
been growth in this system during the years we have observed- Nonuse of 
evaluation was the standard when our study began, although, at that 
time, evaluation had been mandated in the district by federal and local 
school administrators for almost ten years. 
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While there is stiii a predominance or the Routine or lower levels o-f 
use, the data document gradual changes m the ways that prifnary users in 
this district view evaluation and incorporate it into their program 
jnanagement functions. Test score use is a good example of this 
evolution. At the beginning of this study, there ras wide variation m 
the interest in test scores, with many school principals simply shelving 
them as they came, never really certain what "those coaiputer sheets" 
said, and not very interested in finding out. Now, all the principals 
and teachers we interviewed expect tests in the spring and scores in 
September. The scores are sought in chart form so they can be 
distributed at start-up fail meetings. Although the upper levels of use 
are still uncominon, the most successf ul school principals and teachers 
use test data creatively and proactively, along with other supportive 
information about individual children, special groups , and programs 
at the Renewal level. 

This instrument is designed to describe the evaluation use behavior of 
individuals, but it can also be applied to the district level. In our 
sample district. Routine use of evaluation has often been accomplished, 
at least in regard to the basic instructional program in reading and 
mathematics. School years are planned around testing cycles; associate 
superintendents call for accountability plans that include reviewing 
test information and targeted goals that reference test data. Many 
routine meetinos now include the use of surveys and evaluations of how 
well agendas are accomplished. These surveys are usually very simple 
and may neglect to elicit critical evaluations of substance. 
Nevertheless, their use indicates beginning understanding of the ways 
evaluation can contribute to program development. 

In addition to defining the six stages of evaluation use, this draft of 
the Structure of School Evaluation Use also sets out an initial list of 
behavioral indicators — Categories of Use — that describe different 
components of the evaluation use process (Appendix A, Part II). Still 
building on the Texas R D system, we observed seven categories similar 
to those identified by Hall and Loucks (Ibid.). Each category 
describes key actions that are specific to the evaluation use process. 
They are: planning, assessing, acquiring, knowing, managing, 
collaborating, and committing, part II of Appendix A gives examples of 
the behavioral indicators within each category that are characteristic 
of individuals at the highest levels of evaluation use. As this 
research continues, we will return to the original data base to identify 
the additional behaviors within each category that are typical of 
individuals functioning ac each of the six stages. 

Applications of the Structure of School Evaluation Use 

The next steps in the development of the Structure of School Evaluation 
Use are to determine its applicability through pilot testing and to 
refine it for practical use. We submit it at this phase in its 
development for comments and suggest ions from colleagues who have been 
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close to the issues related to the limits oi school evaluation use. We 
believe it can be most helpful to evaluators and decision makers in 
determining the type oi evaluation that would best suit a given school 
program context. 

Training oi evaluators to assess clients' levels of potential ability to 
use the evaluation proces? will increase the likelihood that evaluations 
will be designed and conducted so they more closely meet users' 
abilities and awareness. li staff level coordinators, directors, and 
principals are not able to use evaluation information except at initial 
use levels, only modified and limited evaluation designs are required. 
Evaluators will be in a better position to promote increasingly 
constructive evaluations if thsy can assess the context and work at the 
various developmental stages of their users. Over time, understanding 
and working collaboratively within an individual's readiness to use the 
evaluation process assure that the evaluations in schools will be more 
effectively and continually used. 

To experienced school district evaluators, this structure and the 
observations we make here .ay state the obvious. We have seen that 
most effective LEA evaluators intuitively understand and spontaneously 
use a developmental framework to assess the evaluation context and to 
design their evaluations accordingly (Patton, 1978). Successful 
evaluators know that when project di rectors and evaluators, or 
evaluation directors, superintendents, and school boards work 
collaboratively, in well-matched relationships, they develop mutual 
understandings of how best to incorporate the evaluation process into 
the organization and program structure. The "levels o< evaluation use" 
concept is therefore not new to experienced and effective evaluators, 
but, like much street level knowledge, its reality is only informally 
recognized in the evaluation use literature. 

Evaluation Is Worth the Effort 

This approach to understanding and assessing the process of school 
evaluation use assumes that adopting evaluation processics within school 
districts requires a new way of doing things. As with other 
innovations, integrating evaluation into complex school organizations is 
evolutionary. When the evaluation process becomes fully used, carefully 
established equilibria among individuals and subsystems in the 
organization will inevitably be disrupted, and disruption is never i?asy. 

Although we lament school systems' resistance to the rational use of the 
evaluation process, we have recognized that, in spite of our frustration 
with the pace, the process has been incorporated gradually and in 
piecemeal fashion into many aspects of school life, in the classroon and 
at district levels. Progress is evident if we look back to the 1960s 
when evaluation was first mandated as part of the program implementation 
process. Evaluation units in school districts are increasingly -.ccepted 
components of the school bureaucratic structure; instructional 
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assessoient and standardized testing are routinely expected by children, 
*'eacher5 and ths community; program evaluation training is provided in 
uni'^ersi ties and colleges; and state legislatures are mandating 
evaluation procedures much as did the -federal government in the late 
i960s. Principals, even teachers, now occasionally pick up the phone to 
invits the district's evaluation specialist tn help interpret test 
in-forroation or to design a simple evaluation -for a pet program. Even 
jncre exciting is the increasing number of complex and longitudinal 
evaluation studies rp;.orted each year by school district evaluation 
units. Look, for example, at the annual submissions of district level 
evaluations sponsored by AERA's Division H. The number and quality of 
submissions grow annually. 

The improvements we have seen in evaluation methodology and in the 
technology that supports the evaluation process show the potential value 
of continuing to require that educational programs be evaluated. 
However, we have to become more adaptabls and creative in our use of the 
process. Although bottom line recommendations and changes that respond 
directly to evaluation findings may be the exception, not the rule, a 
broader understanding of the evaluation process and its evolution among 
individual users recognizes that, givsn the opportunity and the proper 
support, the evaluation process will, indeed, become gradually 
structured into educational and program development routines. 

A systematic evaluation model such as the Structure of School Evaluation 
can help in assessing the status of evaluation use in school settings. 
Used diagnostical ly by evaluators, decision makers, trainers, and change 
agents, it can direct planning so that program activities are designed 
to move individuals and the organization towards more complex stages of 
evaluation use. 

Our longitudinal study of evaluation use in process allows us to more 
optimistically answer the questions with which we began. But, to do so 
requires that we recognize the incremental ways that school evaluation 
processes are contributing to the evolution of more rational educational 
decision making. The key is, first, to design mechanisms that assure 
the evaluation use process is implemented routinely into school programs 
and, ultimately, to build into accountability models the expectation and 
requirement that educational systems refine and renew the evaluation 
process continually. 

Looked at with this developmental perspective, recognizing the 
evolutionary nature of the process, we see that the evaluation process 
in schools can be worth the effort and that it falls to the evaluator to 
make it so. 
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APPENDIX A 



THE STRUCTURE OF S'^HOOL 



E.ALUATION 



USE 



(SSEU) 



Ei len M. Pechman 



and Jean A 



King 



(Adapted from 



Hail, 1982 



and Hall h 



Loucks, 



1975) 



Introductory Note: 

The levels of evaluation use defined below assume that users and 
evaluators arc dthicial, motivated by good intentions, and adhere to 
established professional norms and standards. Intentional abuse and 
misuse are independent levels of evaluation use that have been 
documented as part of the evaluation process (Ginsberg, 1984; King ^ 
Pechman, 1982; Lipsky, 1978) and are always potentially a problem. We 
recognize abuse and misuse of the evaluation process as a predictable 
side effect of unreasonable, irrational, or rigid bureaucratic 
expectations and requirements regarding program performance and/or 
evaluation. When "street level" (Lipsky, 1978) staff are called upon to 
demonstrate their program's si:tcess through excessive record keeping or 
arbitrary data collection requirements and without sufficient personal, 
professional, or resource support, evaluation abuse or misuse can easily 
occur (Ginsberg, 1984). 

Evaluation abuse refers to the behaviors of evaluation users that break 
professional norms and use the evaluation process inappropriately. 
Abuse includes actions that are altruistically intended, but misdirected 
»e.g., erasing students' test answers or artificially reducing ^retest 
scores), and those prompted by personal weakness and/or fear. 
Malevolent use of evaluation information has also been observed -ind is 
included in this category. 

The reality of evaluation abuse and misuse must be acknowledged and 
explored, and we should seek the reasons for their occurrence so steps 
can be tal en to minimize such destructive fallout or "side effects" 
(Ginsberg, 1984) of the evaluation process. 
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STRUCTURE QF SCHOOL EVALUATION USE 



Part I; Levels of School Evaluation Use 
0 Nonuse 

User ignores evaluation process or is unaware of its potential 
use and does not seek it out. Evaluation information is not 
integrated into program implementation. 



Decision Point I; User takes action to learn more about the 

requirements and procedures for conducting an evaluation. 



I . Or i ent ati on 

User begins to explore the potential value of the evaluation 
process and tt?kes steps to acquire information about how to 
conduct an evaluation. Orientation may be initiated because 
of a grant or program mandate for an evaluation component in a 
new program; requirements established by a higher level 
manager; or user awareness of the potential value of the 
evaluation process to support, defend, or demonstrate the 
success of an ongoing program. 



Decision Point II; User de ^ ides to initiate the evaluation proces-i by 
c> ^lling for a review of an evaluation design and by taking 
action to determine what must be done to meet the requireflients 
of the evaluation design. 



1 1 , Prepar at i on 

User recognizes the need to collect data and directs staff to 
begin data collection, but remains aloof from the process, 
disregarding it during implementation, except when required to 
approve procedural activities or requi reinents. The evaluation 
is assigned to staff to implement, and data may be included in 
reports, but user is not aware of how to analyze or apply data 
other than as a signal to be reported as mandated. 



Decision Point IIIA; User begins first use of evaluation by reviewing 
data, requesting assistance in interpreting evaluation, or 
including infonnation from evaluation in conversations with 
staff and in staff meetings, reports, or planning,. 
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IIIA, Mechanical Use 

User employs evaluation in required ways e.g., mandated 
reporting, distributing simple summaries of information and 
key data with other program documents. Allows evaluation 
process to proceed, but does not attend to quality of data 
collection process, giving only lip service to the place of 
evaluation in program implementation, planning or change. 



Decision Point IIIB; Routine of evaluation implementation and use is 
established within program. Data collection and simple 
evaluation reporting for signal purposes are regular features 
of annual program implementation timelines and activity, 

IIIB. Routine Use 

User routinely employs the signal -function of the evaluation 
process and reports information regularly at mandated or 
logical points. Program planning and implementation include a 
skeleton evalu^ition process in which simple quantitative 
reports arc submitted at required intervals to higher 
managerial levels and to staff. User fails to implement a 
systematic organizational use of the evaluation process and 
employs the information generated by the process in limited, 
nondynamic ways. 



Decision Point IV; This marks the beginning of charged use. Managers 
change program elements, planning procedures, and/or program 
components on the basis of formal or informal evaluation 
information to improve the program and increase client 
outcomes. 



IV. Ref i nement 

Beginning of charged use, i.e., data are systematically used 
to make program changes. There is an attempt to mold the 
evaluation process to provide information needed for 
program-spocific decisions. The evaluation process is altered 
as needed to meet the client's needs and to assure data are 
available to guide program change."^. Decision makers beg:n to 
seek out usable evaluation information by modifying the 
evaluation design as necessary and by providing staff time and 
personnel to assure that data are systematically generated and 
dependably reported. In short, the decision maker's stated 
commitment to evaluation use is turned into initial action 
(beginn'.ng stages of putting theory into practice). 
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Decision Point V; User initiates use of evaluation information by 

coordinating with staff and/or colleagues to take action for 
program changes that are recommended or implied in the 
recommendations from the evaluators' informal feedback and 
from more formal evaluation data and reports. 



V. Integrati on 

Implements higher levels of charged use by supporting the 
evaluation process both verbally and through managerial 
actions. Evaluation information that is collected is 
carefully determined on the basis of realistic assessments of 
available staff time to collect, analyze and actually use 
information obtained. User carefully collect! data that is 
relevant and accurate and modifies reports ffyr practical use 
by various audiences. Data are routinely disseminated and 
responded to by audiences; colleagues collaborate to assure 
that evaluation information is interpreted, understood and 
distributed. Program changes are monitored and, to the extent 
possible, are made in light of the evaluation findings. 



Decision Point VI; User begins exploring program alternatives or major 
modifications called for by the evaluation and the evaluation 
process; Initiates major planning efforts; recommends 
substantive curricular; budget, or staffing changes on the 
basis of evaluation data; and includes extended and 
**ell -integrated evaluation process as a center-core of the 
program's functioning. 



VI . Renewal 

User continually evaluates the evaluation process, assuring 
that evaluation is the basis of action at every stage, from 
design planning through data definition, collection 
procedures, reporting, and progra/D modification. A delicate 
balance is struck that recognizes practical limitations and 
possibilities and weighs them against bold actions so that the 
'most evaluation for the buck' is obtained from the resources 
that are available. Evaluator or decision maker knows what 
the evaluation process can yield and works to assure that 
information is available and of high quality. User and 
evaluator routinely confer to obtain continual information on 
new developments from the field and in the literature, and 
collaboratively recommend ways to improve program, whether the 
news is good or bad. The evaluation process is comprehensive 
and attempts to reach all levels (top managers through 
front-line staff implementers and audiences/clients). Ideally, 
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at all levels, trofn teacher-child interaction through 
directors to the top manager-director interaction, both 
quantitative and qualitative evaluation data, combined with 
sensitive and practical interpretation — recognizing 
political realities and limitations are the foundations o-f 
actions. 
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STRUCTURE QF SCHOOL EVALUfiTIQN USE 

Part II; Categories of Use and Sample Behavioral Indicators 

The following Categories oi behavioral indicators are based on those 
presented in the Levels oi Use (Hall, et al.,1975). These categories 
are specMic to the evaluation use process and have been developed on 
the basis oi field data from the longitudinal study of school evaluation 
use. The evaluation categories describe the key functions 
characteristic of users at the highest levels of use. 



1: Planning - Designs and outlines short and/or long range steps to be 
taken during process of evaluation, that is, aligns resources, schedules 
activities, and meets with others to organize and/or to coordinate 
evaluation use. This effectively initiates the institutionalization of 
the evaluation process by committing resources and time» 

2: Assessing - Examines potential or actual use of evaluation or some 
aspect 0-^ it; designs the evaluation processes, products, and uses; 
analyzes data;, and conducts neta evaluations. This can be a mental 
assessment or can involve actual collection and analysis of data. 

3: Acquiring- - Solicits information about evaluation in various ways, 
including questioning resource people, corresponding with resource 
agencies, reviewing printed materials, and making visits; collects 
available products; reviews reports; seeks additional data and 
evaluation information as needed* 



4: Knowing -- Refers to the information that the user has about 
evaluation and the evaluation process, how to use it, and consequences 
of its use: awareness of information needs; evaluation potential; how 
to collect, interpret and use data; technical issues; and knowledge of 
the evaluation process. This category refers to cognitive knowledge ^ 
about the evaluation process and about how to use the evaluation / 
information. Feelings or attitudes are not included in this category. 

5: Managi n g (^Hall's 'Performing') - Carries out the actions and 
activities entailed in operationalizing evaluation: puts theory into 
action; uses clout and political context continuously and effectively; 
turns information into action; through effective monitoring and 
directing, teaches others to do same; follows through on implementation 
of evaluation at all levels; and uses organizational roles and functions 
to affect evaluation process. 

6: Col 1 abor at i ng (*Hall's 'Sharing') - Discusses evaluation with 
others* Shares plans, ideas, resources, outcomes, and problems related 
to evaluation use: collaborates with key institutional units and with 
key personnel; reports to others; and holds meetings to exchange 
information and to facilitate use of evaluation results. 
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7: Committing (*Hali'5 Status Reporting) - Describes personal stand in 
relation to evaluation use: demonstrates willingness to take a stand, 
uses in-formation with key audiences and personnel, and advertises 
program evaluation process as key component. 
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APPENDIX B 



LEVELS OF USE OF THE INNOVATION 



Frofli: The LcU Chart > Austin: Research and Developaient Center -for 
Teacher Education, The University o-f Texas at Austin, 1975. 



Dtcision Point A 
Z mZENTATION 

Dtcisiofi Point B 
ZX PXEPAMTIOI 



Statt in which th« uswr hM litUt or oo kaovl«dg« of 
thm innovation, no involvtiMnt with tht ixmovation# 
and it doing nothing toward b«co«ing iAvolvtd* 

Takas action to laam aora datailad infonution 
about tht innovation* 

Stata in which tha vsar has raccntXy acquirtd or is 
acquiring infomation about tha innovation and/or has 
cacantly axplored or is axploring its valua orianta* 
tioo and its dawandt upon usar and usar system. 

Nakaa a dacision to usa tha innovation by astab- 
lishing a tiaa to bagin* 

f tata iA which tha usar ia prtparing for first usa of 
tha innovation* 



Dtcision Point C Changas, if any, and usa ar(« dominatsd by user needs* 

III HECRANZCAL USE State in which the usar focuses aost effort on the 
short-tazsi, dayto*day use of the innovation with 
litUe tisM foi reflection* Changes in use are nade 
aore to aeet user needs than client needs* The user 
is priauurily engaged in a stepwise atttxopt to master 
tha tasks required to use the innovation* often 
resulting in disjointed and superficial use* 

Decision Point 0*1 A routine pattern of use is established* 

ZVA KOUTINE Ose Of the innovation is stabilized* Few, if any« 

changes are being made in ongoing use. Little prep- 
aration or thought is being given to improving inno- 
vation use or its consequences* 

Decision Point D-2 Changes use of the innovation based on formal or 
informal evaluation in order to increase client 
outcootas* 



IVB RCFINDICNT 



State in which the user varies the use of the innova- 
tion to increase the impact on clients within the 
isMdiate sphere of influence. Variations are based 
on knowledge of both short- and long-tem consequences 
for clients* 



Daeiaioa Point X Znitiates changes in usa of innovation based on 
input of and in coordination with what oolleaguas 
are doing* 

V nmBCHATZOH Stata in ^^ch tha usar is coeibining own efforts to 

usa the innovation with related activities of col- 
laaguas to aehiava a coUectiva iapact on clients 
within their c os» ua sphere of iaflttanea* 

Decision Point P Bagins aaploring alternatives to or major modifica- 
tiona of tha innovation presently in use* 

VZ M W CW AXt State in which the user raavaluatas the quality of 

uaa of the innovation, seeks major modifications of 
or altamativas to present innovation to achieve in- 
eraaaad isipact on clients » examinea new developments 
la tha fieldf and axploras new foaXa for aalf and tha 
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